EV: Multilevel Music Knowledge Representation and Programming

This paper introduces EV Meta-Model, a new system for representing musical knowledge for computer-aided composition. It starts with a brief historical discussion on the fields of composition and sound synthesis. Then, the practice of musical composition is presented as a communication process from composers to listeners, where musical messages go through different representations: from the complex abstractions of composers and their compositional tools, to the performers and the perceptual representation of listeners. EV Meta-Model is proposed as a generic tool for representing any kind of time-based events that manifest themselves as coherent representations at different levels, including high-levels of musical abstractions. At the same time, it is intended to be a dynamic representation system, capable of handling each element as a "living" variable, and transmitting such dynamic character to the music that it represents. As examples of its applicability, the paper presents the Evscore Ontology, the implementation of EVcsound, a tool for the creation of detailed compositions with flexible temporal representations, and finally an example of algorithmic composition upon the EV Model. 1. The Representation of Musical Knowledge A suitable knowledge representation tool (KR) is fundamental for a successful Artificial Intelligence system development. Brachman explains the KR concept with these words [Brachman, Levesque, 1985]: “It simply has to do with writing down, in some language or communicative medium, descriptions or pictures that correspond in some salient way to the world or a state of the world". The effectiveness of the intelligent system will depend, to a great extent, on how that KR fits the problem domain. It can therefore be established that the first step for the design of an effective musical composition system is the definition of a musical KR that is appropriated to the domain and to the creative manipulations within the domain. Conventional music notation allows for communication between the composer and the interpreter. But in this case, the compositional knowledge corresponding to the mental abstractions of the composer is not represented explicitly. The compositional techniques, their experience, their creative processes and their intentions are not clearly represented by conventional music notation. In order to study what this compositional knowledge consists of, an analysis of the composition and the creative processes involved should be developed. Figure 1. Subprocesses Cycle in Composition Composition is comprised of subprocesses in the composer abstraction, such as conception, abstraction, imagination, implementation, analysis and correction. Figure 1 is a representation of an analysis model of composition processes. It is shown as a cyclic process of subprocesses. Starting from a voluntary intention or from an emotion, an element is conceived. Following it is imagined, and abstracted in the context from both experience and musical knowledge, and also, why not, from inspiration. Once imagined, the element is concretized, implemented, and tested. Often, the implementation is not explicit, but a mental visualization of the result is enough. Sometimes drawings and sketches are used for testing and analysis. Once evaluated, the affections to the rest of the composition are analyzed. As a consequence, new corrections are conceived starting a new cycle again. Every musical element or structure in the composition could be seen as a product of this kind of synthesisanalysis cycle. In this analytic search of musical knowledge, several components have been identified. They include entities, relationships, procedures, strategies, and metaknowledge. They are explained in table 1. We can conclude that the musical knowledge comprises aspects, elements, procedures and strategies brought into play by the composer during the music creation process. Table 1. Components of Musical Knowledge Elements and entities, Conceptualization and identification of elements and abstract objects arranged into ontological classes. These elements might be considered at different levels, from atomic elements such as notes, to more and more complex structures constructed by combinations of simpler elements. Objects and relationships Rules, patterns, constraints Relationships between the entities above, creating a definition of musical language and style. Intention driven procedures Procedures to develop the musical discourse, the temporal evolution, the narrative line. Rules breaks, originality Exceptions to the rules, patterns, creative innovations at all levels of relationships and structures. Providing liveliness, originality and freshness to the discourse. Strategies Based on experience, they constitute a whole heuristic of composition as a solution search General Criteria, Global Intentions, Finality Criteria above the composition process, as motivation, aesthetics and intention for composing. Some criteria as attention attraction, surprise, equilibrium. Meta-knowledge. 2. Paradigms in Music Representation 2.1.The score paradigm The traditional musical score can be considered as a set of symbols arranged in a temporal succession, representing musical entities such as notes, durations and pitches. All annotations in the score are symbolic representations of instructions indicating what the interpreters should do in order to play the music. In this sense, the score is regarded as a good representation for the performance of music. One could hardly consider musical scores as a comprehensive KR method because information about the composition itself has to be deduced by means of an objective analysis of the score or obtained through subjective interpretation that is part of the listening process. Being the performance instructions specific to different types of musical instrument, the composition is not completely defined without the specification of the instruments and the contribution of the interpreters. Therefore, attaching the corresponding orchestra completes the definition of a composition. Considered from the point of view of its function, an orchestra could be defined as a sound generator that has some knowledge about the interpretation of the score and its translation into acoustic waves. This Orchestra-Score metaphor is implemented in numerous sound synthesis systems, such as MusicV and its descendants Csound [Vercoe, 1994], CLM [Schottstaedt, 1994] and a few others. In Csound, for example, the composition is split into the score file (.sco) and the orchestra file (.orc). 2.2.The unit generator (UG) paradigm The architecture of the first analog synthesizers has greatly influenced the development of sound synthesis systems. Those machines required the physical interconnection of small operating units called unit generators (UG). By means of the combination of UGs, multiple architectures with different performance could be assembled. This philosophy has been traditionally applied in numerous systems for musical composition and sound synthesis. For example, in Csound instruments are programmed using variables to connect basic op-codes building instruments as more complex operating elements. In CLM and Nyquist [Dannenberg, 1997] the interconnection of UGs is defined using the LISP language, and in some other systems, such as PD [Puckette, 1997], these connections are done by means of a graphical interface. From a point of view of KR, those structures could be represented as elements of the type ‘operating-unit-in the-time-domain’, defined as an interconnection of simpler units. It is interesting to note, however, that the UG paradigm allows for the definition of high-level structures starting from similar structures in a lower level. 2.3.The MIDI paradigm MIDI standardization has unquestionably greatly influenced the development of computer music. In music representation, it considers that music is performed with a keyboard. This simplification provides the possibility of representing every musical note just with a number associated with the key being pressed, the pressing velocity, and when and for how long it is pressed. MIDI has been useful as a music representation for the last decades, but it cannot be considered as a complete representation system, as it does not include any information concerning the orchestra. Although some attempts have been made to standardize some orchestras (e.g., General MIDI), the impossibility of unifying MIDI players and it low resolution, prevent MIDI from being an efficient tool for representing musical sound. However, it provides the possibility of developing systems with low computational costs. Within algorithmic composition, systems such as Common Music (CM) [Taube, 2004] or Symbolic Composer (SCOM) [Stone, 1997] are examples of MIDI-based systems implemented in LISP. 3. From Composer To Listener We consider the musical phenomenon as a communication process starting from the abstractions and emotions of the composer towards the ear of the listeners. Figure 2 shows the different representations that musical messages go through from the abstract world level, down to the acoustic level. The acoustic level can be reached through two possible ways. In the traditional way, a human interpreter plays the conventional score found in the notation level. Composers write the score from mental abstractions. Usually, compilation does not take place directly, but sketches, scripts, plan drawings, drafts and many other methods are used. This is a long elaboration process where composers repeatedly go back to higher levels to perform adjustments and make finer conceptualizations of their abstractions. It is an iterative and feedback debugging process, where many different paths are tried out and their results evaluated. From the point of view of Artificial Intelligence, this process could be considered as heuristic searches for the satisfaction of self-imposed constraints. Figure 2. Levels of representation in musical communication In conclusion, it is a time consuming work developed at levels above the level of the interpretation languages. The other way of producing sound comes from a stratum immediately above, the audio world. A speaker, reproducing audio samples, now emits sound. Sound files are created based on performance events at the interpretation level like MIDI, Csound . sco or any other event list created by the composer. 3.1.The metalevel paradigm The interactive composition process described for the creation of the score is similarly applicable in this case. The metalevel is not seen here just as a layer in the levels of structure, but as a stratified zone capable of supporting different architectures. In that area, we could place the following: • Newell’s "knowledge level" [Newell-1982], corresponding to the symbolic level of the score, where the musical entities of score are situated. • Conceptualizations of musical elements in the dimensions of time and form, not directly represented in the score, such as motives, sequences, form structures and their relationships. • Extra-score conceptualizations in the spectral or pitch dimension, such as intervals, chords, harmonic elements, tonalities and so on. • New "musical meta-objects" at closer to composer levels. • Elements and procedures of algorithmic musical generation. The metalevel is an attractive area for the development of compositional tools. It is, however, necessary to use an efficient KR that is flexible and able to cope with new entities at several levels, and which can integrate new musical elements and structures created by the composer. Music created with computers is often regarded by the general public as mechanical and lifeless. Note that in conventional music communication, both the composer and the interpreter contribute their human dimension to the music, In this case, the final sound carries creative contributions and “beautiful imperfections”, which manifest the richness and liveliness of its human origin. This is a target not to be forgotten when designing a representation system for musical composition. 4. EV Meta-Model: a Multilevel Music Representation EV Meta-Model is proposed as a KR for elements in time at different levels. One of the key points of this approach is its simplicity. Figure 3 shows the core of our ontology, consisting on three main classes: the event, the parameter and the dynamic object. The definition of the classes is given as follows (in LISP notation): (define-Class EVENT :is-a evclass :slots ((start-position :type real) (length :type real) (parameters :multiple list :type parameter) (events :multiple list :type event) (position-function :type function))) (define-Class PARAMETER :is-a evclass :slots ((name :type string) (value :type dynamic-object))) (define-Class DYNAMIC-OBJECT :is-a evclass :slots ((type :doc "Type of the out value") (out :type t) (dyn-function :type function) (dyn-arguments :multiple list :type DYNAMIC-OBJECT) (status-memory :type t :doc "State memory") )) Figure 3. Ontological Core of EV Meta-Model.